Skip to content

[fix][schema] Reject unsupported Avro schema types during schema registration #24103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Apr 17, 2025
Merged

[fix][schema] Reject unsupported Avro schema types during schema registration #24103

merged 9 commits into from
Apr 17, 2025

Conversation

poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Mar 21, 2025 •
edited by lhotari
Loading

Motivation

Issue

  • User can upload an Avro-Union schema by pulsar-admin API
  • However, the schema uploaded can not be used, users will get an error "Not a record" when creating a producer with AUTO_PRODUCE schema.

Modifications

  • Deny uploading an Avro-Union schema
  • Improve the logs when creating a producer failed due to an invalidated schema

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode added type/bug The PR fixed a bug or issue reported a bug release/4.0.4 release/3.0.11 release/3.3.6 labels Mar 21, 2025
@poorbarcode poorbarcode added this to the 4.1.0 milestone Mar 21, 2025
@poorbarcode poorbarcode self-assigned this Mar 21, 2025
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 21, 2025
@poorbarcode poorbarcode requested review from lhotari and removed request for lhotari March 26, 2025 01:22
@poorbarcode poorbarcode requested review from lhotari and removed request for lhotari March 26, 2025 10:03
lhotari
lhotari approved these changes Mar 26, 2025
View reviewed changes
Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, good work @poorbarcode

@lhotari lhotari changed the title [fix][schema]Pulsar does not support avro-union schema, but it allows it to be registered [fix][schema] Reject unsupported Avro schema types during schema registration Mar 26, 2025
@lhotari
Copy link
Member

lhotari commented Mar 26, 2025

I renamed the title to "Reject unsupported Avro schema types during schema registration" since good PR titles are optimally expressed in imperative mood, like good commit messages. The PR title shouldn't be a description of the problem it's fixing. It might be better to file a separate issue where that could be a title of the issue itself.

@poorbarcode
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@poorbarcode
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@lhotari
Copy link
Member

lhotari commented Apr 9, 2025 •
edited
Loading

There's a test failure:

  Error:  Tests run: 175, Failures: 1, Errors: 0, Skipped: 165, Time elapsed: 46.633 s <<< FAILURE! - in org.apache.pulsar.broker.admin.AdminApiSchemaTest
  Error:  org.apache.pulsar.broker.admin.AdminApiSchemaTest.testInvalidSchemaDataException[V1](4)  Time elapsed: 0.01 s  <<< FAILURE!
  java.lang.AssertionError: expected [422] but found [500]
  	at org.testng.Assert.fail(Assert.java:110)
  	at org.testng.Assert.failNotEquals(Assert.java:1577)
  	at org.testng.Assert.assertEqualsImpl(Assert.java:149)
  	at org.testng.Assert.assertEquals(Assert.java:131)
  	at org.testng.Assert.assertEquals(Assert.java:1418)
  	at org.testng.Assert.assertEquals(Assert.java:1382)
  	at org.testng.Assert.assertEquals(Assert.java:1428)
  	at org.apache.pulsar.broker.admin.AdminApiSchemaTest.testInvalidSchemaDataException(AdminApiSchemaTest.java:282)
  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  	at java.base/java.lang.Thread.run(Thread.java:1583)

It seems that 422 Unprocessable Content status is expected.

@github-actions github-actions bot added the PIP label Apr 15, 2025
@lhotari
Copy link
Member

lhotari commented Apr 16, 2025

@poorbarcode It seems that this test failure remainns:

  Error:  Tests run: 41, Failures: 1, Errors: 0, Skipped: 17, Time elapsed: 151.005 s <<< FAILURE! - in org.apache.pulsar.schema.SchemaTest
  Error:  org.apache.pulsar.schema.SchemaTest.testSendAvroAndJsonPrimitiveSchema  Time elapsed: 0.187 s  <<< FAILURE!
  org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: {"errorMsg":"Please call org.apache.pulsar.client.api.Schema.INT when using a simple type schema","reqId":3649047843048571735, "remote":"localhost/127.0.0.1:45669", "local":"/127.0.0.1:38284"}
  	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1067)
  	at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:107)
  	at org.apache.pulsar.schema.SchemaTest.testSendAvroAndJsonPrimitiveSchema(SchemaTest.java:408)
  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  	at java.base/java.lang.Thread.run(Thread.java:1583)

@poorbarcode
Copy link
Contributor Author

poorbarcode commented Apr 17, 2025 •
edited
Loading

@lhotari

@poorbarcode It seems that this test failure remainns:

Sorry, the schema typed Avro & Primitive is supported, I have removed the check of these schemas and added a new test to ensure no break changes in the future: testAvroIntSchema

@lhotari lhotari merged commit 3bdc661 into apache:master Apr 17, 2025
51 of 52 checks passed
lhotari pushed a commit that referenced this pull request Apr 17, 2025
@poorbarcode @lhotari
[fix][schema] Reject unsupported Avro schema types during schema regi…
033727c 
…stration (#24103)

(cherry picked from commit 3bdc661)
lhotari pushed a commit that referenced this pull request Apr 17, 2025
@poorbarcode @lhotari
[fix][schema] Reject unsupported Avro schema types during schema regi…
523c5c8 
…stration (#24103)

(cherry picked from commit 3bdc661)
lhotari pushed a commit that referenced this pull request Apr 17, 2025
@poorbarcode @lhotari
[fix][schema] Reject unsupported Avro schema types during schema regi…
0e4ca80 
…stration (#24103)

(cherry picked from commit 3bdc661)
@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.22%. Comparing base (bbc6224) to head (cdfeff7).
Report is 1028 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #24103      +/-   ##
============================================
+ Coverage     73.57%   74.22%   +0.65%     
+ Complexity    32624    32505     -119     
============================================
  Files          1877     1865      -12     
  Lines        139502   144603    +5101     
  Branches      15299    16505    +1206     
============================================
+ Hits         102638   107337    +4699     
+ Misses        28908    28787     -121     
- Partials       7956     8479     +523
Flag Coverage Δ
inttests 26.75% <31.57%> (+2.16%) ⬆️
systests 23.24% <57.89%> (-1.08%) ⬇️
unittests 73.72% <100.00%> (+0.87%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../service/schema/validator/SchemaDataValidator.java 80.00% <100.00%> (+1.05%) ⬆️
...ce/schema/validator/StructSchemaDataValidator.java 83.33% <100.00%> (+7.14%) ⬆️
.../client/impl/schema/generic/GenericSchemaImpl.java 82.35% <100.00%> (+5.42%) ⬆️

... and 1072 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lhotari lhotari lhotari approved these changes

@congbobo184 congbobo184 congbobo184 approved these changes

Assignees

@poorbarcode poorbarcode

Labels
cherry-picked/branch-3.0 cherry-picked/branch-3.3 cherry-picked/branch-4.0 doc-not-needed Your PR changes do not impact docs PIP ready-to-test release/3.0.12 release/3.3.7 release/4.0.5 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
4.1.0
Development

Successfully merging this pull request may close these issues.
4 participants
@poorbarcode @lhotari @codecov-commenter @congbobo184